Databases and file management programs are well known facilities by means of which data generated by or requested by an application program for a computer can be stored in a predefined reliable manner in non-volatile storage, such as disk storage. Such data management facilities spare the application programmer from having to write his own data management code and offer reliable storage of data through defined access methods.
One of the best known file management programs is the Virtual Storage Access Method (VSAM) developed by IBM. VSAM is available as part of several IBM Operating Systems, such as MVS. ("IBM" AND "MVS" are trademarks of International Business Machines Corporation.) The most common way of managing files in VSAM is by the use of Key Sequential Data Sets (KSDS) in which records may be accessed by means of a key (such as an employee name) and an index which contains pointers to the physical record location.
In order to store data in a VSAM KSDS, the program must define the average and maximum lengths of the records in the data set. This is in order to optimize data I/O performance having regard to the likely size of records and frequency of access.
The data portion of a record is first preassembled in the main memory of the computer with a unique key and then stored with the key, in an area of non-volatile memory such as disk storage, allocated to the particular data set. The appropriate index is updated so that the particular data record can be retrieved.
Further detail about VSAM can be found in the book "Getting into VSAM--An Introduction and Technical Reference" by Michael P Bouros (2nd Ed., John Wiley & Sons, 1987).
It can happen that, in other aspects of its operation, an application program may create very long records in main memory (for example, graphic image data) which it then wishes to store in a data set or data base. Aspects of handling long fields in a recoverable data base are discussed in EPA 0336548-A2. One problem that is not discussed is that such long records may exceed the permitted record length for the data set or data base and thus can only be stored if they are split into a sequence of physical records, each of which has a key consisting of a common portion and a unique differentiating portion.
A naive approach to splitting a long record would be to copy a chunk of the long record into an intermediate buffer, to generate and add the appropriate key and then to store the preassembled key and data in non-volatile storage. The process would then be repeated for successive chunks of the long record, each time assembling the KSDS record in an intermediate buffer in main memory. This is wasteful and inefficient in that two moves of each chunk of data are necessary before it is stored in non-volatile storage.